Skip to content

Conversation

@butofleury
Copy link
Member

@butofleury butofleury commented Oct 1, 2025

What problem is this PR solving? Explain here in one sentence.

Related JIRA tickets : WC2-817

Self proofreading checklist

  • Did I use eslint and ruff formatters?
  • Is my code clear enough and well documented?
  • Are my typescript files well typed?
  • New translations have been added or updated if new strings have been introduced in the frontend
  • My migrations file are included
  • Are there enough tests?
  • Documentation has been included (for new feature)

Doc

Tell us where the doc can be found (docs folder, wiki, in the code...).

Changes

  • Added a filter on last updated_at based to the latest date when ETL task run successfully.
  • Generating analytics data only for entities based their latest updated date.

How to test

From a local instance with wfp coda database having South Sudan and/or Ethiopia and/or Nigeria data run ETL script to generate data and Celery configured, try to identify latest success task for:

  • Update the date_created for latest task for each task_name 'plugins.wfp.tasks.etl_ethiopia', 'plugins.wfp.tasks.etl_ng', 'plugins.wfp.tasks.etl_ssd'
  • South Sudan: select id,task_name,status,date_created from django_celery_results_taskresult where status='SUCCESS' and task_name='plugins.wfp.tasks.etl_ssd' order by id desc limit 1

  • Nigeria: select id,task_name,status,date_created from django_celery_results_taskresult where status='SUCCESS' and task_name='plugins.wfp.tasks.etl_ng' order by id desc limit 1

  • Ethiopia: select id,task_name,status,date_created from django_celery_results_taskresult where status='SUCCESS' and task_name='plugins.wfp.tasks.etl_ethiopia' order by id desc limit 1

Try to update the date_created for each of the 3 tasks.

Then run the script to ETL for all data : docker compose run iaso manage etl_ssd && docker compose run iaso manage etl_eth && docker compose run iaso manage etl_ng

It will generate analytics data for entities with the updated at = date_created.

Print screen / video

Upload here print screens or videos showing the changes.

Notes

Things that the reviewers should know:

  • known bugs that are out of the scope of the PR
  • other trade-offs that were made
  • does the PR depends on a PR in bluesquare-components?
  • should the PR be merged into another PR?

Follow the Conventional Commits specification

The merge message of a pull request must follow the Conventional Commits specification.

This convention helps to automatically generate release notes.

Use lowercase for consistency.

Example:

fix: empty instance pop up

Refs: IA-3665

Note that the Jira reference is preceded by a line break.

Both the line break and the Jira reference are entered in the Add an optional extended description… field.

@madewulf
Copy link
Member

madewulf commented Oct 3, 2025

@butofleury I think this PR should also include a way to re-run the ETL on the whole data so that we can re-run it easily if an error is identified in the computation

@beygorghor beygorghor removed their request for review October 10, 2025 09:39
Copy link
Member

@quang-le quang-le left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see @madewulf 's comment

@beygorghor beygorghor added the release Should be released in production at next deploy label Oct 20, 2025
@madewulf madewulf merged commit eaad02b into develop Oct 21, 2025
3 checks passed
@madewulf madewulf deleted the WC2-817-Optimise-memory-usage-of-celery-task branch October 21, 2025 13:15
@tdethier tdethier added user tested Has already been tested on staging ok for release Released and removed ok for release labels Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release Should be released in production at next deploy Released user tested Has already been tested on staging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants